Camera view and screen scraping for information extraction from imaging scanner consoles

ABSTRACT

An apparatus provides remote assistance to a local operator of a medical imaging device disposed in a medical imaging device bay via a communication link from a remote location that is remote from the medical imaging device bay to the medical imaging device bay. The apparatus includes a workstation disposed at the remote location including at least one workstation display. At least one electronic processor is programmed to, over the course of a medical imaging examination performed using the medical imaging device: extract successive image frames from video or screen sharing of a controller display of the medical imaging device; screen-scrape information related to the medical imaging examination from the successive image frames over the course of the medical imaging examination; maintain status information on the medical imaging examination at least in part using the screen-scraped information; and output an alert perceptible at the remote location when the status information on the medical imaging examination satisfies an alert criterion.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.17/097,306 filed Nov. 13, 2020, which claims the benefit of U.S.Provisional Patent Application No. 62/936,741 filed Nov. 18, 2019. Theseapplications are hereby incorporated by reference herein.

FIELD

The following relates generally to the imaging arts, remote imagingassistance arts, remote imaging examination monitoring arts, and relatedarts.

BACKGROUND

Medical imaging, such as computed tomography (CT) imaging, magneticresonance imaging (MRI), positron emission tomography (PET) imaging,fluoroscopy imaging, and so forth, is a critical component of providingmedical care, and is used in a wide range of medical fields, such ascardiology, oncology, neurology, orthopedics, to name a few. Theoperator of the medical imaging device used to acquire the medicalimages is typically a trained technician, while interpretation of themedical images is often handled by a medical specialist such as aradiologist. Interpretation of radiology reports or findings by theradiologist can be handled by the patient's general practitioner (GP)physician or another medical specialist such as a cardiologist,oncologist, orthopedic surgeon, or so forth.

The operator of a medical imaging device of a given modality (CT, MRI,PET, etc.) is often expected to be qualified to perform a wide range ofdifferent imaging procedures. For example, a cardiac imaging proceduremay be very different from an imaging procedure targeting a known orsuspected oncological tumor, which may be very different again from anorthopedic imaging procedure, etc. This requires the imaging deviceoperator to be highly qualified, and preferably experienced, in adiverse range of different types of imaging procedures. Furthermore, theincreased demand for medical imaging services has led to most hospitalsproviding medical imaging departments, and additional independentimaging laboratories that provide services on a contractual basis. Thishas led to strong demand for highly qualified and experienced medicalimaging device operators.

Currently, a local technologist performs most of the steps in theimaging workflow, which can include preparing the patient, reviewing theprocedure with the patient, reviewing the screening form, getting thepatient changed, inserting the IV as needed, positioning the patient inthe scanner, performing the scan, and reformatting the images. Dependingon the experience of the technologist and complexity of the exam, he orshe may require assistance from a senior technologist in positioning thepatient, carrying out the scan, reviewing image quality, uncoveringsources of artefacts, etc. Unfortunately, depending on the institution,the talent pool of its technologists, or the time of the exam, thesenior technologist may not be available to help with every case whereassistance is required. This may result in a scan that deviates from astandard protocol and quality standards, making diagnosis and potentialfollow-up challenging, or a scan that is insufficient for diagnosticinterpretation by a radiologist, in which case, a patient will have tocome back for a repeat scan at the expense of the imaging center, whichis costly for the radiology provider.

As the demand for imaging studies is forecast to continue to grow, itwould be beneficial to provide junior technologists with on-demandguidance and supervision for complex scans or technical issues. Remoteradiology operations command centers have been considered to addressthis problem, by establishing channels of access to skilled seniortechnologists, who may be cost-effectively based at a single remotecommand center that services multiple medical institutions.

The following discloses certain improvements to overcome these problemsand others.

SUMMARY

In one aspect, an apparatus provides remote assistance to a localoperator of a medical imaging device disposed in a medical imagingdevice bay via a communication link from a remote location that isremote from the medical imaging device bay to the medical imaging devicebay. The apparatus includes a workstation disposed at the remotelocation including at least one workstation display. At least oneelectronic processor is programmed to, over the course of a medicalimaging examination performed using the medical imaging device: extractsuccessive image frames from video or screen sharing of a controllerdisplay of the medical imaging device; screen-scrape information relatedto the medical imaging examination from the successive image frames overthe course of the medical imaging examination; maintain statusinformation on the medical imaging examination at least in part usingthe screen-scraped information; and output an alert perceptible at theremote location when the status information on the medical imagingexamination satisfies an alert criterion.

In another aspect, a non-transitory computer readable medium storesinstructions executable by at least one electronic processor forperforming a method of providing remote assistance to a plurality oflocal operators of one or more medical imaging devices disposed in acorresponding number of medical imaging device bays via a communicationlink from a remote location that is remote from the medical imagingdevice bays to the medical imaging device bays. The method includes,over the course of medical imaging examinations using the medicalimaging devices: extracting successive image frames from video or screensharing of controller displays of the medical imaging devices;screen-scraping information related to the medical imaging examinationsfrom the successive image frames over the course of the medical imagingexaminations; maintaining status information on the medical imagingexaminations at least in part using the screen-scraped information; andoutputting an alert perceptible at the remote location when the statusinformation on one or more of the medical imaging examinations satisfiesan alert criterion.

In another aspect, a method for providing remote assistance to a localoperator of a medical imaging device disposed in a medical imagingdevice bay via a communication link from a remote location that isremote from the medical imaging device bay to the medical imaging devicebay includes, over the course of a medical examination: extractingsuccessive image frames from video or screen sharing of a controllerdisplay of the medical imaging device; screen-scraping informationrelated to the medical imaging examination from the successive imageframes over the course of the medical imaging examinations; maintainingstatus information on the medical imaging examination at least in partusing the screen-scraped information; and displaying the statusinformation on the medical imaging examination at a workstation display.

One advantage resides in providing remote operators or radiologists withsituational awareness of local imaging examination(s) which facilitatesproviding effective assistance to one or more local operators atdifferent facilities.

Another advantage resides in providing remote operators or radiologistswith awareness of local imaging examination(s) workflow whichfacilitates providing assistance to a one or more local operators atdifferent facilities.

Another advantage resides in providing a remote operator withinformation about already-performed steps in a workflow in order toprovide assistance for subsequent steps in the workflow.

Another advantage resides in providing a remote operator or radiologistwith status information about an imaging examination based on what isdisplayed on the controller display of the medical imaging device.

Another advantage resides in providing a remote operator or radiologistwith status information on a medical imaging examination using astandard display format that is independent of the controller display ofthe medical imaging device performing the medical imaging examination.

A given embodiment may provide none, one, two, more, or all of theforegoing advantages, and/or may provide other advantages as will becomeapparent to one of ordinary skill in the art upon reading andunderstanding the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure may take form in various components and arrangements ofcomponents, and in various steps and arrangements of steps. The drawingsare only for purposes of illustrating the preferred embodiments and arenot to be construed as limiting the disclosure.

FIG. 1 diagrammatically shows an illustrative apparatus for providingremote assistance in accordance with the present disclosure.

FIG. 2 shows an example flow chart of operations suitably performed bythe apparatus of FIG. 1 .

DETAILED DESCRIPTION

The following relates to a remote radiology operations command centerservice, in which a remotely based senior technician (i.e. “super-tech”)is provided with a camera view of a scanner and console rooms along withscreen sharing of the console display, and can provide telephonic orother assistance to the local technician during challenging imagingexaminations. The super-tech can be expected to be simultaneouslyhandling a number of different imaging bays, and hence cannot becontinuously monitoring any one imaging examination. This could create aproblem if the super tech is unaware of critical information whenturning attention to a particular examination. Another problem is thatthe super-tech may not be monitoring a particular examination during atime-critical event during which the super-tech's assistance would bevaluable. The following assists the super-tech by providing thesuper-tech with status information about the ongoing imagingexamination. The following can be additionally or alternatively employedto assist a radiologist who is tasked with performing peri-examinationimage review by providing the radiologist with status information aboutthe ongoing imaging examination.

In some embodiments disclosed herein, screen scraping of the consoledisplay (and possibly or other equipment displays) can be leveraged toautomatically record dynamic situational awareness, and to detect andissue alerts of events that should be brought to the super-tech's (orradiologist's) attention. Screen content may be scraped from the sharedscreen or from frames of a video camera whose field of view includes thescreen. Using the shared screen as the source for the screen scrapingcan be more accurate compared with screen scraping of video, as a videocamera view may be occasionally occluded by movement of personnel andmay have low resolution; however, not all equipment displays willnecessarily be screen-shared with the technician, in which case acamera-based screen scraping is suitable.

The scraped screen is analyzed to identify relevant information on thecurrent status of the imaging examination, such as imaging settings orimaging sequence parameters, any alerts shown on the screen, textuallabels of importance such as a textual label of the type of imagingexamination being performed, and so forth. The console user interface(UI) presented on the controller display of the medical imaging devicetypically has a number of different dialog screens, such as examinationor scan selection dialog screen(s), scan settings dialog screen(s),acquisition monitoring dialog screen(s), and so forth, each of whichcontain different types of information. The screen scraping is thereforeperformed continuously to detect when new dialog screens are accessed bythe local technician and to extract the information contained in eachaccessed dialog screen, and to detect when (for example) the value of aparticular scan setting is updated by the local technician and extractthe updated value.

The presentation of the status information on the medical imagingexamination obtained (at least in part) from the screen scraping may bepresented at the remote radiology operations command center (and/or at acomputer used by a reviewing radiologist) can optionally bevendor-agnostic, that is, the status information on the medical imagingexamination can be displayed at the workstation display at the remoteradiology operations command center (or radiologist's workstation) usinga standard display format that is independent of the controller display(which is vendor-specific). Optionally, this standardization may extendto performing suitable unit conversion or the like so that all valuesobtained by the screen scraping are presented in a uniform format. Forexample, a time-based value may be presented by one vendor inmillisecond units and by another vendor in second units, and thestandardization may include converting to a vendor-agnostic choice ofunits.

To perform the screen scraping, a template of the set of dialog screensof the UI for the specific imaging modality and vendor/model of theimaging system is stored, and the appropriate template is used toidentify the locations and content of areas of the screen (i.e.“fields”) for each displayed dialog screen. Use of such vendor-,modality-, and model-specific templates enables efficient screenscraping for a wide range of medical imaging devices, thereby enablingthe system to be vendor-agnostic and useable with any imaging device forwhich an appropriate template is available. Additionally oralternatively, optical character recognition (OCR) performed on thescraped screen can be used to identify (possibly vendor-specific) labelsannotated to various types of information. For example, in a portion ofan MRI settings dialog screen containing “TE=

ms” the value may be interpreted as “time-to-echo” of 12 millisecondsbased on the labels “TE” which represents “time-to-echo” and “ms” whichindicates milliseconds. A vendor/model/modality-specific database oflabels can be maintained, so that the display is correctly interpretedin cases in which different vendors (or models et cetera) use differentlabels for the same value.

In other embodiments disclosed herein, to improve efficiency of thesuper-tech, as previously noted, the scraped information can bedisplayed at the super-tech's computer in the remote center using astandardized status display that is the same for a given imagingmodality regardless of the particular vendor. This facilitates thesuper-tech's ability to comprehend the information without needing tomentally adapt to the display format of the particular vendor. However,as knowledge of the vendor/make/model of the imaging system may beimportant information for the super-tech, the standard display mayprovide vendor/make/model information in a designated field of thestandardized status display. Whenever the super-tech selects to monitora given examination, the current standardized status display is broughtup, so that the super-tech can be quickly brought up to date with thestate of the examination.

In some embodiments disclosed herein, the system further includesworkflow templates for various imaging examination workflows, optionallyalong with standard values for those workflows where available. Theappropriate workflow template is retrieved based on information scrapedfrom the console screen, or based on information otherwise provided tothe remote service center (e.g., based on a schedule of examinationsbeing supervised by the super-tech). Alert criteria are applied to thescraped information to detect critical events such as extensive editingof scan settings by the local technician, alerts or warnings or errormessages issued on the console, multiply repeated imaging sequences, animaging examination that is running excessively long, or so forth. Whena critical event is detected, a human-perceptible (e.g. visual and/oraudible) alert is issued to the super-tech.

In other embodiments disclosed herein, the current standardized statusdisplay may be used in other contexts. For example, if an on-callradiologist is to review images during the imaging examination, then thestandardized status display may be presented to the radiologist alongwith the images so that the radiologist is appraised of the imagingexamination status. Similarly, certain generated alerts might beadditionally or alternatively sent to the on-call radiologist if thealert is of a type that could be effectively handled by the radiologist.

With reference to FIG. 1 , an apparatus for providing assistance from aremote operator RO (or super-tech) to a local operator LO is shown. Asshown in FIG. 1 , the local operator LO, who operates a medical imagingdevice (also referred to as an image acquisition device, imaging device,and so forth) 2, is located in a medical imaging device bay 3, and theremote operator RO is disposed in a remote service location or center 4.It should be noted that the “remote operator” RO may not necessarilydirectly operate the medical imaging device 2, but rather providesassistance to the local operator LO in the form of advice, guidance,instructions, or the like. The remote location 4 can be a remote servicecenter, a radiologist's office, a radiology department, and so forth.The remote location 4 may be in the same building as the medical imagingdevice bay 3 (this may commonly be the case, for example, in the case ofa “remote operator” RO who is a radiologist tasked with peri-examinationimage review), but more typically the remote service center 4 and themedical imaging device bay 3 are in different buildings, and indeed maybe located in different cities, different countries, and/or differentcontinents. In general, the remote location 4 is remote from the imagingdevice bay 3 in the sense that the remote operator RO cannot directlyvisually observe the imaging device 2 in the imaging device bay 3 (henceoptionally providing a video feed as described further herein).

The image acquisition device 2 can be a Magnetic Resonance (MR) imageacquisition device, a Computed Tomography (CT) image acquisition device;a positron emission tomography (PET) image acquisition device; a singlephoton emission computed tomography (SPECT) image acquisition device; anX-ray image acquisition device; an ultrasound (US) image acquisitiondevice; or a medical imaging device of another modality. The imagingdevice 2 may also be a hybrid imaging device such as a PET/CT orSPECT/CT imaging system. While a single image acquisition device 2 isshown by way of illustration in FIG. 1 , more typically a medicalimaging laboratory will have multiple image acquisition devices, whichmay be of the same and/or different imaging modalities. For example, ifa hospital performs many CT imaging examinations and relatively fewerMRI examinations and still fewer PET examinations, then the hospital'simaging laboratory (sometimes called the “radiology lab” or some othersimilar nomenclature) may have three CT scanners, two MRI scanners, andonly a single PET scanner. This is merely an example. Moreover, theremote service center 4 may provide service to multiple hospitals. Thelocal operator controls the medical imaging device 2 via an imagingdevice controller 10. The remote operator is stationed at a remoteworkstation 12 (or, more generally, an electronic controller 12).

As used herein, the term “medical imaging device bay” (and variantsthereof) refer to a room containing the medical imaging device 2 andalso any adjacent control room containing the medical imaging devicecontroller 10 for controlling the medical imaging device. For example,in reference to an MRI device, the medical imaging device bay 3 caninclude the radiofrequency (RF) shielded room containing the MRI device2, as well as an adjacent control room housing the medical imagingdevice controller 10, as understood in the art of MRI devices andprocedures. On the other hand, for other imaging modalities such as CT,the imaging device controller 10 may be located in the same room as theimaging device 2, so that there is no adjacent control room and themedical bay 3 is only the room containing the medical imaging device 2.In addition, while FIG. 1 shows a single medical imaging device bay 3,it will be appreciated that the remote service center 4 (and moreparticularly the remote workstation 12) is in communication withmultiple medical bays via a communication link 14, which typicallycomprises the Internet augmented by local area networks at the remoteoperator RO and local operator LO ends for electronic datacommunications.

As diagrammatically shown in FIG. 1 , a camera 16 (e.g., a video camera)is arranged to acquire a video stream 17 of a portion of the medicalimaging device bay 3 that includes at least the area of the imagingdevice 2 where the local operator LO interacts with the patient, andoptionally may further include the imaging device controller 10. Thevideo stream 17 is sent to the remote workstation 12 via thecommunication link 14, e.g. as a streaming video feed received via asecure Internet link. Additionally, a screen mirroring data stream 18 issent from the imaging device controller 10 to the remote workstation 12.The communication link 14 also provides a natural language communicationpathway 19 for verbal and/or textual communication between the localoperator and the remote operator. For example, the natural languagecommunication link 19 may be a Voice-Over-Internet-Protocol (VOIP)telephonic connection, an online video chat link, a computerized instantmessaging service, or so forth. Alternatively, the natural languagecommunication pathway 19 may be provided by a dedicated communicationlink that is separate from the communication link 14 providing the datacommunications 17, 18, e.g. the natural language communication pathway19 may be provided via a landline telephone.

FIG. 1 also shows, in the remote service center 4 including the remoteworkstation 12, such as an electronic processing device, a workstationcomputer, or more generally a computer, which is operatively connectedto receive and present the video 17 of the medical imaging device bay 3from the camera 16 and to present the screen mirroring data stream 18 asa mirrored screen. Additionally or alternatively, the remote workstation12 can be embodied as a server computer or a plurality of servercomputers, e.g. interconnected to form a server cluster, cloud computingresource, or so forth. The workstation 12 includes typical components,such as an electronic processor 20 (e.g., a microprocessor), at leastone user input device (e.g., a mouse, a keyboard, a trackball, and/orthe like) 22, and at least one display device 24 (e.g. an LCD display,plasma display, cathode ray tube display, and/or so forth). In someembodiments, the display device 24 can be a separate component from theworkstation 12. The display device 24 may also comprise two or moredisplay devices, e.g. one display presenting the video 17 and the otherdisplay presenting the shared screen of the imaging device controller 10generated from the screen mirroring data stream 18. Alternatively, thevideo and the shared screen may be presented on a single display inrespective windows. The electronic processor 20 is operatively connectedwith a one or more non-transitory storage media 26. The non-transitorystorage media 26 may, by way of non-limiting illustrative example,include one or more of a magnetic disk, RAID, or other magnetic storagemedium; a solid state drive, flash drive, electronically erasableread-only memory (EEROM) or other electronic memory; an optical disk orother optical storage; various combinations thereof, or so forth; andmay be for example a network storage, an internal hard drive of theworkstation 12, various combinations thereof, or so forth. It is to beunderstood that any reference to a non-transitory medium or media 26herein is to be broadly construed as encompassing a single medium ormultiple media of the same or different types. Likewise, the electronicprocessor 20 may be embodied as a single electronic processor or as twoor more electronic processors. The non-transitory storage media 26stores instructions executable by the at least one electronic processor20. The instructions include instructions to generate a graphical userinterface (GUI) 28 for display on the remote operator display device 24.

The medical imaging device controller 10 in the medical imaging devicebay 3 also includes similar components as the remote workstation 12disposed in the remote service center 4. Except as otherwise indicatedherein, features of the medical imaging device controller 10, whichincludes a local workstation 12′, disposed in the medical imaging devicebay 3 similar to those of the remote workstation 12 disposed in theremote service center 4 have a common reference number followed by a“prime” symbol, and the description of the components of the medicalimaging device controller 10 will not be repeated. In particular, themedical imaging device controller 10 is configured to display a GUI 28′on a display device or controller display 24′ that presents informationpertaining to the control of the medical imaging device 2, such asconfiguration displays for adjusting configuration settings an alert 30perceptible at the remote location when the status information on themedical imaging examination satisfies an alert criterion of the imagingdevice 2, imaging acquisition monitoring information, presentation ofacquired medical images, and so forth. It will be appreciated that thescreen mirroring data stream 18 carries the content presented on thedisplay device 24′ of the medical imaging device controller 10. Thecommunication link 14 allows for screen sharing between the displaydevice 24 in the remote service center 4 and the display device 24′ inthe medical imaging device bay 3. The GUI 28′ includes one or moredialog screens, including, for example, an examination/scan selectiondialog screen, a scan settings dialog screen, an acquisition monitoringdialog screen, among others. The GUI 28′ can be included in the videofeed 17 or the mirroring data stream 17′ and displayed on the remoteworkstation display 24 at the remote location 4.

The remote workstation 12 is configured as described above to perform amethod or process 100 for providing assistance the local operator LO.The method or process 100 includes a screen sharing process 101 whichperforms screen scraping on the controller display 24′ of the medicalimaging device 2, and the method or process 100 includes maintainingstatus information on the ongoing medical imaging examination based oninformation obtained by the screen scraping 101. The screen scrapingprocess 101 operates to extract (from the video 17 or the screen sharing18) successive image frames of the controller display 24′ of the medicalimaging device 2, and to screen-scrape information related to themedical imaging examination from the successive image frames over thecourse of the medical imaging examination. The non-transitory storagemedium 26 stores instructions which are readable and executable by theat least one electronic processor 20 (of the workstation 12, as shown,and/or the electronic processor or processors of a server or servers ona local area network or the Internet) to perform disclosed operationsincluding performing the method or process 100. In some examples, themethod 100 may be performed at least in part by cloud processing. Inparticular, the GUI 28 presented on the display 24 of the remoteworkstation 12 preferably includes a window presenting the video 17, anda window presenting the mirrored screen of the medical imaging devicecontroller 10 constructed from the screen mirroring data stream 18, andstatus information on the medical imaging examination that is maintainedat least in part using the screen-scraped information. This allows theremote operator RO to be aware of the content of the display of themedical imaging device controller 10 (via the shared screen) and also tobe aware of the physical situation, e.g. position of the patient in themedical imaging device 2 (via the video 17), and to additionally beaware of the status of the imaging examination as summarized by thestatus information. During an imaging procedure, the natural languagecommunication pathway 19 is suitably used to allow the local operator LOand the remote operator RO to discuss the procedure and in particular toallow the remote operator to provide advice to the local operator.

With reference to FIG. 2 , and with continuing reference to FIG. 1 , anillustrative embodiment of the method 100 is diagrammatically shown as aflowchart. The method 100 is performed over the course of a medicalimaging examination performed using the medical imaging device 2. At anoperation 102, the at least one electronic processor 20 of theworkstation 12 in the remote location 4 is programmed to receive atleast one of: video 17 from the video camera 16 of the medical imagingdevice 2 located in the medical imaging device bay 3; and/or the screensharing 18. The video feed 17 and/or the screen sharing 18 can bedisplayed at the remote workstation display 24, typically in separatewindows of the GUI 28.

At an operation 104, the at least one electronic processor 20 of theworkstation 12 in the remote location 4 is programmed to extractsuccessive image frames from the video 17 or from the screen sharing 18of the controller display 24′ of the medical imaging device 2. Theextracted successive frames may not necessarily include every imageframe of the acquired video or screen sharing. For example, the video orscreen sharing might be at 30 frames per sec (fps), but the screenscraping might only be able to process 5 fps due to computationallimitations. In that case, the successive image frames might be everysixth video frame. The extracted successive frames are bitmap images.These bitmap images may be color or monochrome images, depending on thesource video 17 or screen sharing 18 and, in the case of color video orscreen sharing, depending on whether the extraction operation 104retains the color information. The extracted successive frames containinformation such as details of the imaging protocol being acquired,aborted or repeated image sequences, an amount of time left in themedical exam, error messages generated by the imaging device 2, patientbreathing, heart rate waveforms, a patient screening form, and so forth.However, the information is contained in bitmap image form, and hence isnot readily machine interpretable.

Accordingly, at an operation 106, the at least one electronic processor20 of the workstation 12 in the remote location 4 is programmed toscreen-scrape information related to the medical imaging examinationfrom the extracted successive image frames over the course of themedical imaging examination (although this operation may take place inthe medical device imaging bay 3 and transmitted to the remotelocation). The extraction and screen scraping operations 104, 106correspond to the screen-scrapping process 101 of FIG. 1 . At anoperation 108, the at least one electronic processor 20 is programmed tomaintain status information on the medical imaging examination at leastin part using the screen-scraped information. As diagrammaticallyindicated by looping arrow 109, the operations 102, 104, 106, 108 arerepeated in iterative fashion over the course of a medical imagingexamination performed using the medical imaging device 2 in order tomaintain real time current status information generated at operation108. During a routine imaging examination, or during routine portions ofan imaging examination, the maintained status information will indicatethe imaging examination is progressing favorably, and is unlikely tobenefit from intervention by the remote operator RO. This isadvantageous, because in many practical situations the remote operatorRO may be tasked with assisting a number of ongoing imaging examinationsconcurrently, and cannot continuously monitor all of the simultaneousimaging examinations tasked to the remote operator RO. However, themaintained status information can provide information indicating thatthere is a problem with the imaging examination, or may detect someother situation such as a warning displayed on the controller display24′, which indicates that intervention by the remote operator RO may beadvantageous. Accordingly, at an operation 110, the at least oneelectronic processor 20 of the workstation 12 in the remote location 4is programmed to output an alert 30 perceptible at the remote locationwhen the status information on the medical imaging examination satisfiesan alert criterion (or, some other alert criterion is met, such asdetecting a warning displayed on the controller display 24′ via thescreen scraping 101).

The information screen-scraped in the operation 106 from the imageframes extracted in the operation 104 can include a variety ofinformation. Although described below in terms of a single imagingdevice bay 3, the at least one electronic processor 20 can beiteratively executing a plurality of instances of the operations 104,106 to screen-scrape video 17 (or screen-sharing 18) of a correspondingplurality of medical imaging devices 2 disposed at a correspondingnumber of medical imaging devices bays 3.

The operations 106-110 can be performed in a variety of manners, notnecessarily being mutually exclusive. In one example, the screenscraping of information 106 detects when the successive image framesdepict a change to a new dialog screen on the GUI 28′ presented on thecontroller display 24′. That is, the local operator LO is operating themedical imaging device 12 using the dialog screens on the GUI 28′. TheGUI 28′ is captured in the successive image frames. The screen-scrapingoperation 106 then includes screen-scraping at least a portion of theinformation from one or more of the successive image frames that depictthe new dialog screen.

In another example, the screen scraping of information 106 detects a newsetting value for the medical imaging examination in the successiveimage frames. That is, the local operator LO adjust a setting value onthe GUI 28′ of the controller display 24′. The GUI 28′, with thisupdated setting value, is captured in the successive image frames. Thescreen-scraping operation 106 then includes screen-scraping the newsetting value from one or more imaging frames of the successive imageframes that depict the new setting value. The maintaining operation 108includes updating the status information on the medical imagingexamination on the workstation display 24 with the screen-scraped newsetting value.

In a further example, the screen scraping of information 106 identifiesone or more dialog screens in the GUI 28′ of the controller display 24′in the successive image frames. The workstation 12 then retrieves adialog screen template 32 from a database stored on the non-transitorycomputer readable medium 26. The template 32 corresponds to a dialogscreen depicted in in one or more of the image frames, and the templateis usually one of a set of dialog screen templates for the GUI 28′ ofthe vendor/make/model of the medical imaging device 2. By providingdialog screen template sets for various GUIs of various imaging devicesavailable from a range of vendors, the system can operate in conjunctionwith imaging devices of a wide range of different imaging modalitiesmade by different vendors. The template 32 identifies one or more screenregions in the GUI 28′, and associates the one or more screen regionswith settings of the medical imaging examination. The screen-scrapingoperation 106 includes screen-scraping information from an image frameunder analysis by the remote operator RO in one or more screen regions.The screen-scraped information is then associated with settings of themedical imaging examination using the associations provided by thecorresponding dialog screen template 32.

In yet another example, one of the image frames under analysis by thescreen scraping of information 106 is screen-scraped using OCR toextract a value and one or more labels proximate to the value in theimage frame under analysis. The remote workstation 12 then identifies acurrent setting value for the medical imaging examination by associatingthe screen-scraped value with a setting of the medical imagingexamination based on the screen-scraped one or more labels that arepositioned proximate to the value in the image frame under analysis. Forexample, if the screen-scraped portion of the image frame includes asetting “TE=120 ms”, the value may be interpreted as “time-to-echo” of120 milliseconds based on the labels “TE” which represents“time-to-echo” and “ms” which indicates milliseconds. The maintainingoperation 108 then includes updating the status information on themedical imaging examination on the workstation display 24 with thescreen-scraped current setting value.

In another example, one of the image frames under analysis by the remoteoperator RO is screen-scraped to detect an error or warning messagedepicted on the controller display 24′ of the medical imaging device 2.The output operation 110 includes outputting an alert 30 perceptible atthe remote location 4 (e.g., visually displayed on the workstationdisplay 24, audibly output with a speaker (not shown), and so forth)when the detected error or warning messages satisfies an error orwarning alert criterion. In one approach, the set of templates 32include templates for most or all error or warning messages that can beissued by the GUI 28′ of the imaging device 2, and the detection of theerror or warning message entails matching the screen scraped error orwarning message with the template. Such matching could be done in imagespace, where the error or warning message templates 32 are images, orcould be done after OCR screen scrapes the text of the error or warningmessage, in which case the matching entails matching the OCR'd error orwarning text with textual templates 32.

In yet a further example, the screen-scraping operation 106 includesperforming OCR processes on an image frame under analysis by the remoteoperator RO to extract textual information.

The maintaining operation 108 looped over the course of the imagingexamination as indicated by a loop 109, operates to maintain currentstatus information on the medical imaging examination at least in partusing the screen-scraped information. A portion of the statusinformation may also be obtained from other sources, such as from directmachine-readable communication with the device controller 10 of themedical imaging device 2. In some embodiments, the status information isdisplayed on the workstation display 24, along with the screen-scrapedinformation. For example, the status information can be displayed usinga standard display format that is independent of the controller display24′. The standard display format can include a display of a vendor ofthe medical imaging device 2, along with optionally including a make andmodel of the medical imaging device.

At the operation 110, the alert 30 is output when the status informationon the medical imaging examination satisfies an alert criterion. To doso, in one example, the status information is compared with informationin one or more workflow templates 32 retrieved based on the informationscreen-scraped in the operation 106. The alert 30 can be output whenthis comparison of the status information with the information in thetemplates 32 satisfies the alert criterion (e.g., an echo time in thestatus information being above or below a threshold listed in thetemplate(s)). In another example, setting values shown in the statusinformation are compared with corresponding values in the template(s)32, and the alert 30 is output when one or more of the values is outsidea range of corresponding values in the information of the template (s)32. In a further example, the alert 30 is output when the alertcriterion based on the status information indicates significant changesin the imaging examination, such as extensive editing of scan settingson the GUI 28′ by the local operator LO, the local operatormultiple-repeating imaging sequences, an imaging session runningexcessively long, and so forth.

The disclosure has been described with reference to the preferredembodiments. Modifications and alterations may occur to others uponreading and understanding the preceding detailed description. It isintended that the exemplary embodiment be construed as including allsuch modifications and alterations insofar as they come within the scopeof the appended claims or the equivalents thereof.

1. A non-transitory computer readable medium storing instructionsexecutable by at least one processor for performing a method forproviding remote assistance to a local operator of a medical imagingdevice disposed in a medical imaging device bay which is remote to aworkstation display, the method comprising, over the course of a medicalimaging examination: extracting successive image frames from video orscreen sharing of a controller display of the medical imaging device;screen-scraping information related to one or more settings of themedical imaging device used in the medical imaging examination from thesuccessive image frames over the course of the medical imagingexamination, the one or more settings of the medical imaging deviceincluding one or more of an examination dialog screen, a scan selectiondialog screen, a scan settings dialog screen, or an acquisitionmonitoring dialog screen; maintaining status information of the medicalimaging examination at least in part using the screen-scrapedinformation; and displaying the status information of the medicalimaging examination at the workstation display.
 2. The non-transitorycomputer readable medium according to claim 1, wherein screen-scrapinginformation includes: detecting when the successive image frames depicta change to a new dialog screen presented on the controller display; andscreen-scraping at least a portion of the information from one or moreof the successive image frames that depict the new dialog screen.
 3. Thenon-transitory computer readable medium according to claim 1, whereinscreen-scraping information includes: detecting a new setting value forthe medical imaging examination in the successive image frames; andscreen-scraping the new setting value from one or more imaging frames ofthe successive image frames that depict the new setting value; whereinthe status information on the medical imaging examination is updatedwith the screen-scraped new setting value.
 4. The non-transitorycomputer readable medium according to claim 1, wherein screen-scrapinginformation includes: identifying a corresponding dialog screen templatethat corresponds to a dialog screen depicted in an image frame underanalysis of the successive image frames wherein the corresponding dialogscreen template identifies one or more screen regions and associates theone or more screen regions with settings of the medical imagingexamination; and screen-scraping information from the image frame underanalysis in the one or more screen regions and associating thescreen-scraped information in the one or more screen regions withsettings of the medical imaging examination using the associationsprovided by the corresponding dialog screen template.
 5. Thenon-transitory computer readable medium according to claim 1, whereinscreen-scraping information includes: screen scraping an image frameunder analysis of the successive image frames to extract a value and oneor more labels that are positioned proximate to the value in the imageframe under analysis; and identifying a current setting value for themedical imaging examination by associating the screen-scraped value witha setting of the medical imaging examination based on the screen-scrapedone or more labels that are positioned proximate to the value in theimage frame under analysis; wherein the status information on themedical imaging examination is updated with the current setting value.6. The non-transitory computer readable medium according to claim 1,wherein screen-scraping information includes: screen scraping an imageframe under analysis of the successive image frames to detect an erroror warning message depicted on the display screen of the medical imagingdevice; wherein the at least one processor is further programmed tooutput an alert to the workstation display when the detected error orwarning message satisfies an error or warning alert criterion.
 7. Thenon-transitory computer readable medium according to claim 1, whereinscreen-scraping information includes: performing optical characterrecognition (OCR) on an image frame under analysis of the successiveimage frames to extract textual information.
 8. The non-transitorycomputer readable medium according to claim 1, wherein the screensharing of the controller display of the medical imaging device sharesthe screen at the workstation display.
 9. The non-transitory computerreadable medium according to claim 1, wherein the at least one processoris further programmed to: receive a video feed capturing the controllerdisplay of the medical imaging device; and display the video feed at theworkstation display; wherein the extraction of the successive imageframes comprises extraction of the successive image frames from thereceived video feed.
 10. The non-transitory computer readable mediumaccording to claim 9, wherein the at least one processor is programmedto display images acquired by the medical imaging device during themedical imaging examination at the workstation display.
 11. Thenon-transitory computer readable medium according to claim 1, whereinthe at least one processor is further programmed to: display the statusinformation on the medical imaging examination at the workstationdisplay.
 12. The non-transitory computer readable medium according toclaim 11, wherein the at least one processor is programmed to displaythe status information on the medical imaging examination at theworkstation display using a standard display format that is independentof the controller display.
 13. The non-transitory computer readablemedium according to claim 12, wherein the standard display formatfurther includes a display of at least a vendor of the medical imagingdevice.
 14. The non-transitory computer readable medium according toclaim 1, wherein the at least one processor is programmed to: retrieveat least one workflow template based on the information extracted fromthe screen-scraped video; compare the status information on the medicalimaging examination with information in the retrieved at least oneworkflow template; and wherein the alert criterion is based on thecomparison of the status information on the medical imaging examinationwith the information in the retrieved at least one workflow template.15. The non-transitory computer readable medium according to claim 14,wherein the at least one processor is programmed to: compare the statusinformation on the medical imaging examination with corresponding valuesof the information in the at least one workflow template; and whereinthe alert criterion is based on the status information on the medicalimaging examination being outside a range of values of the informationin the at least one workflow template.
 16. The non-transitory computerreadable medium according to claim 14, wherein the alert criterion isbased on the maintained status information on the medical imagingexamination indicating one or more of: extensive editing of scansettings by the local operator; multiply-repeated imaging sequences;and/or an imaging session running excessively long.
 17. Thenon-transitory computer readable medium according to claim 1, whereinthe at least one processor is programmed to screen-scrape video of aplurality of medical imaging devices disposed at a corresponding numberof medical imaging device bays.
 18. A method for providing remoteassistance to a local operator of a medical imaging device disposed in amedical imaging device bay which is remote to a workstation display, themethod comprising, over the course of a medical imaging examination:extracting successive image frames from video or screen sharing of acontroller display of the medical imaging device; screen-scrapinginformation related to one or more settings of the medical imagingdevice used in the medical imaging examination from the successive imageframes over the course of the medical imaging examination, the one ormore settings of the medical imaging device including one or more of anexamination dialog screen, a scan selection dialog screen, a scansettings dialog screen, or an acquisition monitoring dialog screen;maintaining status information of the medical imaging examination atleast in part using the screen-scraped information; and displaying thestatus information of the medical imaging examination at the workstationdisplay.
 19. The method according to claim 18, wherein screen-scrapinginformation includes: detecting when the successive image frames depicta change to a new dialog screen presented on the controller display; andscreen-scraping at least a portion of the information from one or moreof the successive image frames that depict the new dialog screen. 20.The method according to claim 19, wherein screen-scraping informationincludes: detecting a new setting value for the medical imagingexamination in the successive image frames; and screen-scraping the newsetting value from one or more imaging frames of the successive imageframes that depict the new setting value; wherein the status informationon the medical imaging examination is updated with the screen-scrapednew setting value.
 21. The method according to claim 18, whereinscreen-scraping information includes: performing optical characterrecognition (OCR) on an image frame under analysis of the successiveimage frames to extract textual information.
 22. The method according toclaim 18, wherein screen-scraping information includes: screen scrapingan image frame under analysis of the successive image frames to detectan error or warning message depicted on the display screen of themedical imaging device; wherein the at least one processor is furtherprogrammed to output an alert to the workstation display when thedetected error or warning message satisfies an error or warning alertcriterion.
 23. The method according to claim 18, wherein the at leastone processor is programmed to screen-scrape video of a plurality ofmedical imaging devices disposed at a corresponding number of medicalimaging device bays.
 24. The method according to claim 18, wherein theat least one processor is further programmed to: display the statusinformation on the medical imaging examination at the workstationdisplay.